Set up for a Methodological Review Board in Psychology at the University of Padova

L. Sità, M. Manente, A. Perugini, G. Calignano, E. Toffalini, I. Alfarone, T. Marci, M. Pastore, G. Altoè

Methodological Review Board (MRB)

Why the need for a MRB?

  • Need for a cultural shift towards better research practices, starting from our department 1

  • Lack of a review process in most pre-registrations

  • Increasing demand for transparency and Open Science

Project Goals

Establish a sustainable MRB by:

  • Supporting clarity and transparency in pre-registration

  • Offering advisory methodological feedback

  • Maintaining impartiality (e.g., MRB members will not be listed as co-authors)

  • Encouraging Open Science practices

MRB characteristics

  • Voluntary submission to the MRB

  • Advisory only, not gate-keeping

  • Diverse expert reviewers (Statisticians, Psychometricians, Experts of the specific Psychological field)

  • Not a tailored revision of the project, more of a quality check (i.e., sample size justification, clear hypothesis, planned data analysis)

The MRB Workflow

  1. Authors identify an appropriate pre-registration template (e.g. OSF)

  2. Authors complete the chosen template and submit it to the MRB

  3. Review and Feedback (multiple rounds if needed)

  4. Pre-registration Submission

  5. Final Check and Badge Award (if standards are met)

Long-Term Impact Goals

  • Monitor change in pre-registration rates among published studies over time

  • Success will be defined as exceeding a 10% pre-registration rate among eligible studies within the first two years

  • Promoting a culture of preregistration, transparency, open science and and fostering better research practices

The Survey

Department insight

  • Attitudes toward preregistration and Open Science

  • Perceived usefulness and feasibility of MRB

  • Differences between Early Career Researchers (ECRs) and Tenure Track faculty members.

Response rate

Total respondents: 53 out of 142 (representing approximately 37% of the department’s academic staff).

  • Tenure-track: 31 out of 65

  • ECRs: 22 out of 77

BIAS?

Have you ever heard about the replicability crisis?

Have you ever heard about the replicability crisis?

If so, do you think it’s a problem for scientific progress?

If so, do you think it’s a problem for scientific progress?

How much do you think those are the causes?

How much do you think those are the causes?

How much do you think those are the causes?

How much do you think those are the causes?

How much do you think those are the causes?

How much do you think those are the causes?

How much do you think those are the causes?

How much do you think those are the causes?

How much do you think those are the causes?

How much do you think those are the causes?

How much do you think those are the causes?

How much do you think those are the causes?

How much do you think those are the causes?

How much do you think those are the causes?

Have you ever heard of those practices?

Have you ever heard of those practices?

Have you ever heard of those practices?

Have you ever heard of those practices?

Have you ever heard of those practices?

Have you ever heard of those practices?

Have you ever used those practices?

Have you ever used those practices?

Have you ever used those practices?

Have you ever used those practices?

Have you ever used those practices?

Have you ever used those practices?

Have you ever used those practices?

Have you ever used those practices?

Have you ever used those practices?

Have you ever used those practices?

Have you ever used those practices?

Have you ever used those practices?

Can those practices contribute to the progress of the discipline?

Can those practices contribute to the progress of the discipline?

Can those practices contribute to the progress of the discipline?

Can those practices contribute to the progress of the discipline?

Can those practices contribute to the progress of the discipline?

Can those practices contribute to the progress of the discipline?

If you decided to pre-register a study, would you need help?

If you decided to pre-register a study, would you need help?

If you decided to pre-register a study, would you feel more confident after consulting the MRB?

If you decided to pre-register a study, would you feel more confident after consulting the MRB?

How much would a MRB be useful for:

How much would a MRB be useful for:

How much would a MRB be useful for:

How much would a MRB be useful for:

How much would a MRB be useful for:

How much would a MRB be useful for:

How much would a MRB be useful for:

How much would a MRB be useful for:

How much would a MRB be useful for:

How much would a MRB be useful for:

To wrap up

  • ECRs tended to attribute greater importance to the possible causes of the replication crisis

  • Most respondents were familiar with Open Science practices

  • Open materials and open data were used more frequently than pre-registration of hypotheses or analyses.

  • Overall, most researchers felt that these practices are important

  • Not everyone expressed a need for a MRB, but many indicated interest and potential willingness to engage with it.

Time for Feedback!

  • How could we improve people’s interest in the MRB?

  • What incentives can be given to reviewers?

  • Should going through the MRB be mandatory?

  • If so, how can it be implemented without researchers feeling threatened?

PYMS June 5th 2025 - ambra.perugini@phd.unipd.it